Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Initial Codebase for Integrations CLI #1488

Closed
wants to merge 41 commits into from
Closed

Add Initial Codebase for Integrations CLI #1488

wants to merge 41 commits into from

Conversation

Swiddis
Copy link
Collaborator

@Swiddis Swiddis commented Apr 3, 2023 •
edited
Loading

Description

Per issue #1451, this PR implements the basic initial setup for the integrations CLI. From the help page:

Usage: integrations-cli.py [OPTIONS] COMMAND [ARGS]...

  Create and maintain Integrations for the OpenSearch Integrations Plugin.  

Options:
  --help  Show this message and exit.

Commands:
  check    Analyze the current Integration and report errors.
  create   Create a new Integration from a specified template.
  package  Package the current integration for use in OpenSearch.

It has some robustness to be added after the API is ready, particularly with regard to generating and validating sub-components such as saved objects. The current implementation only generates a configuration and references the local schema files.

The PR also creates a CI workflow that tests the new Python code.

TODO before converting from draft:

  • Documentation
  • More robust error handling
  • General code cleanup

Issues Resolved

Part of progress on #1451. This issue can safely be archived and superseded by smaller-scope issues.
Conflicts with #1454.

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@Swiddis
Move schema folder to resources
5253511 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Move schema documentation to docs
43e374b 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Move rest of schema documentation to docs
18d71a7 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Pull CLI from 2.x
ea46eb9 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Commit CLI project to Poetry
472508d 
Poetry is an alternative package manager for Python that is more
scalable. It's used in various large Python projects to good
effect. https://python-poetry.org/

Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Use returns Result for validation
3c11c47 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Pylint maintenance
1766927 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add catalog selection support
d0cf5c1 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Tweak catalog schema to pass validation for catalog
5d86173 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Refactor validation logic
9265e58 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Reformat
15f7a13 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add poetry to CI
d5d1dd1 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Merge remote-tracking branch 'upstream/main' into cli-main
861d573
@Swiddis
Refresh poetry lock
99c72e9 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis Swiddis added enhancement New feature or request ci labels Apr 3, 2023
@Swiddis
Add install to poetry steps
b921d94 
Signed-off-by: Simeon Widdis <[email protected]>
@YANG-DB YANG-DB added this to the 2.8 milestone Apr 3, 2023
YANG-DB
YANG-DB previously approved these changes Apr 3, 2023
View reviewed changes
Copy link
Member

@YANG-DB YANG-DB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a README.md file at the root of the integration-cli package

@Swiddis
Fix catalog schema
7592d2f 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add better component handling
74747bb 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Modify template http dashboard
baf1edb 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Modify dashboards and add upload script
00cc117 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Fix update.sh
f3dfc4c 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Fix broken dashboard
6e4ea43 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Switch from poetry
9775001 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add helper script and update readme
c144a2b 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Update readme with workflow
6801fd5 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Copy link
Collaborator Author

Swiddis commented Apr 12, 2023 •
edited
Loading

Updated TODO list:

  • General Code Clean-up
    • Work on reworking usage of returns (first experiment was sloppy)
    • Fix up error handling
    • Break commands into smaller units for testing
  • Error handling in clear non-happy-path cases
  • Fix the CI pipeline
  • Add more testing
  • Switch to Pytest for consistency with other OS CLI tools
  • Convert to a proper Python package
  • Actual dependency management and validation
    • Detect when components contain other components and add them
    • Validate if subcomponents are missing
  • Improve catalog searching functionality
  • Proper automatic integration uploading

@Swiddis
Fix shell scripts and make upload dynamic
c4a482f 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Switch to Pytest
082a3fc 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Tweak CI
693cc82 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Fix CI attempt
c1b917d 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Use CI from SQL cli
10971b3 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Switch to lru_cache for compatibility
abd0692 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add integrations to branch
67ef7d6 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Handle get-mapping failures
86209e3 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Simplify test docker
0f7a629 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add work on adding subcomponent validation to integrations
1a178f3 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Revert "Add work on adding subcomponent validation to integrations"
cdc62a7 
This reverts commit 1a178f3.
It starts work on new functionality when the functionality depends on other stuff.
Revert to a clean version.

Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Add dependency management and clean handling of unnecessary fields
fc1836e 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Remove dependence on nginx
92d9917 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Setting path doesn't set url
f6a8385 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Fix zipping
81ea966 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis
Remove testing integrations folder
bbe3232 
Signed-off-by: Simeon Widdis <[email protected]>
@Swiddis Swiddis closed this Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YANG-DB YANG-DB YANG-DB left review comments

@ps48 ps48 Awaiting requested review from ps48 ps48 will be requested when the pull request is marked ready for review ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm will be requested when the pull request is marked ready for review kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho will be requested when the pull request is marked ready for review derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 will be requested when the pull request is marked ready for review joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen will be requested when the pull request is marked ready for review dai-chen is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq rupal-bq will be requested when the pull request is marked ready for review rupal-bq is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric will be requested when the pull request is marked ready for review mengweieric is a code owner

@vamsi-amazon vamsi-amazon Awaiting requested review from vamsi-amazon vamsi-amazon will be requested when the pull request is marked ready for review vamsi-amazon is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo will be requested when the pull request is marked ready for review penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az will be requested when the pull request is marked ready for review seankao-az is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha will be requested when the pull request is marked ready for review anirudha is a code owner

Assignees
No one assigned
Labels
ci enhancement New feature or request
Projects
Integration
Status: Done
Milestone
2.8
Development

Successfully merging this pull request may close these issues.
2 participants
@Swiddis @YANG-DB